Recent endeavors mainly focus on parameter efficient transfer learning (PETL) for VLP models by only updating a small number of parameters.

It is interesting to realize scalable neural networks in an architecture by introducing shallow classifiers. Actually, the motivation is not new as some recent work also investigated a similar objective [C1, C2, C3] by the anytime property. However, there are no analyses and comparison with the related studies which should be introduced and compared. The proposed framework is not that significant as the additional components are just borrowed from existing well-developed studies (attention, distillation) as well as the framework requires more parameters and computations. So the methodology itself is still incremental.

Recent endeavors mainly focus on parameter efficient transfer learning (PETL) for VLP models by only updating a small number of parameters. In this paper, we aim at parameter and computation efficient transfer learning (PCETL) for VLP models. In particular, PCETL not only needs to limit the number of trainable parameters in VLP models, but also to reduce the computational redundancy during inference, thus enabling a more efficient transfer. To approach this target, we propose a novel dynamic architecture skipping (DAS) approach towards effective PCETL. Instead of directly optimizing the intrinsic architectures of VLP models, DAS first observes the significances of their modules to downstream tasks via a reinforcement learning (RL) based process, and then skips the redundant ones with lightweight networks, i.e. adapters, according to the obtained rewards.

In this paper, we propose a novel parameter and computation efficient tuning method for Multi-modal Large Language Models (MLLMs), termed Efficient Attention Skipping (EAS). Concretely, we first reveal that multi-head attentions (MHAs), the main computational overhead of MLLMs, are often redundant to downstream tasks. Based on this observation, EAS evaluates the attention redundancy and skips the less important MHAs to speed up inference. Besides, we also propose a novel propagation-of-information adapter (PIA) to serve the attention skipping of EAS and keep parameter efficiency, which can be further re-parameterized into feed-forward networks (FFNs) for zero-extra latency. To validate EAS, we apply it to a recently proposed MLLM called LaVIN and a classic VL pre-trained model called METER, and conduct extensive experiments on a set of benchmarks. The experiments show that EAS not only retains high performance and parameter efficiency, but also greatly speeds up inference speed. For instance, LaVIN-EAS can obtain 89.98\% accuracy on ScineceQA while speeding up inference by 2.2 times to LaVIN

Paula Parpart's research explores why sometimes simpler algorithms can outperform more complex algorithms. Since the 1970s, a rare point of agreement between Nobel Laureate Daniel Kahneman and prominent Max Planck director Gerd Gigerenzer has been that decision heuristics are an alternative to Bayesian rationality. In cognitive science and psychology, heuristics are decision making algorithms that follow a set of simple rules and deliberately ignore information in the input data. For example, when making real-world decisions such as choosing which coffee to buy or choosing which apartment to rent, there are potentially thousands of features that could play into the decision, but we usually do not have the time or memory capacity to use them all. In choosing between two apartments, instead of considering all available information sources such as proximity to work, proximity to schools, crime rates, neighbourhood sport facilities or market trends, a simple heuristic called "Take-The-Best" (Gigerenzer & Goldstein, 1996) would just rely on the first most important cue that is able to discriminate among the apartments, and ignore all other cues.